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VALIDATION, VERIFICATION, AND TESTING 
FOR THE INDIVIDUAL PROGRAMMER 



Martha Branstad 
John C. Chernlavsky 
W. Ri chard s Adrlon 



Guidelines are given for program testing and 
verification to Insure quality software for the 
programmer working alone in a computing environ- 
ment V7ith limited resources. The emphasis is on 
verification as an Integral part of the software 
development ^ Guidance Includes developing and 
planning testing as well as the application of 
other verification techniques at each lifecycle 
stage. Relying upon neither automated tools nor 
formal quality assurance support, the guidelines 
should be appropriate for applications programmers 
doing small development projects . 

Key words: Testing; Program Verification; Software 
Development . 

1. Introduction 



Testing, validation and verification of software is a 
difficult and arduous task even for a manager with automated 
tools and sufficient people to devote to the task. This re- 
port is intended as a guideline to those who are developing 
programs and software with limited resources. It is aimed 
at the production of quality software through the use of 
sound verification methods, techniques, and planning. The 
most common resource limited environment is that of the sin- 
gle programmer working alone on a development project. 

The domain of the solitary programmer is unique and 
warrants a specialized guide to verification and testing. 
Neither the problems nor the solutions that exist for the 
development of medium or large computer systems apply. 
Problems of coordinating several programmers do not exist in 
this domain nor do massive integration efforts. All manage- 
ment is done by the programmer* No independent internal or 
external qual it y assurance groups exist. In addition, the 
problem is usually of a size which is intellectually manage- 
abl e . 
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The verification and validation approaches used for 
medium and large systems are not always applicable to the 
environment of the single programmer. For instance, tool 
development is stressed for large and medium systems. For 
very small program development , time and cost restrictions 
do not allow for speclall zed tool development . Although 
this report assumes limited resources, tools which exist lo- 
cally should not be ignored. A skilled craftsman uses the 
best tools available. 



This report concentrates on inexpensive verification 
and testing techniques to assure the quality of work. For a 
more comprehensive treatment of verification techniques and 
tools the reader is referred to a forthcoming NBS Special 
Publication [ADRI80] , 

Since we bel ieve veri f i cati on should be performed at 
every development stage, this report is organized around the 
life cycle stages given in Figure 1. 



: REQUIREMENTS : 


DESIGN 


: CONSTRUCTION 


OPERATION & : 








MAINTENANCE : 



Figure 1. Life Cycle Stages for Software Development 



Section 2 introduces verification techniques and planning 
which should be employed throughout the life cycle. Section 
3 discusses testing in general and why it is difficult. 
Section 4,5,6, and 7 discuss verification and testing during 
the requirements, design, construction, and operation 
stages, respectively. Section 8 summarizes and presents a 
collection of general rules. 

The following deflnlLlons of common terms are used in 
this document. It should be noted that some of these terms 
t:ay appear with slightly different meanings elsewhere in the 
1 it erat ure . _ . 

1. VALIDATION: determination of the correctness of the 
final program or software produced from a develop-- 
ment project with respect to the user needs and re- 
quirements. Validation is usually accomplished by 
verifying each stage of the software development 11- 
f ecycle . 

2. CERTIFICATION: acceptance of software by an author- 
l7:ed agent usually after the software has been vali- 
dated by the agent, or after its validity has been 
demonstrated to the agent . 
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3. VERIFICATION: In general the demonstration of con- 
sistency, completeness , and correctness of the 
software at each stage and between each stage of the 
development llfecycle. Requirements are verified- 
successive Iterations of the design specifications 
are verified Internally, both against the require- 
ments and the earlier Iterations; modules are unit 
tested and verified against the design specifica- 
tions; and the system undergoes Intensive verifica- 
tion during Integration at the construction stage. 
For small programs, verification Is the process of 
determining the correspondence between a program and 
Its specification. Static anal ysis , dynamic 
analysis, and program testing are used during verif- 
ication. 

4. TESTING: examination of the behavior of a program by 
executing the program on sample data sets. 

5. PROOF OF CORRECTNESS: use of - technl ques of logic to 
Infer that an assertion assumed true at program en- 
try Implies that an assertion holds at program exit. 

6. PROGRAM DEBUGGING: the process of correcting syntac- 
tic and logical errors detected during coding. De- 
t)ugglng occurs before the program or module Is fully 
executable and continues until tha programmer feels 
the program or module is sound and executable. Test- 
ing Is then used to exercise the code over a suffi- 
cient range of test data to verify Its adherence to 
the design and requirements specifications. 

2. Verification Through the Life Cycle 

Problem solving Is a creative process that follows a 
general paradigm. The first step Is to carefully define the 
problem to be solved. Next a solution Is hypothesized. The 
trial solution Is then scrutinized and exercised to deter- 
mine If it works. Based upon the results of the third step, 
modifications are made to the problem statement or to the 
trial solution. The revised solution Is again exercised and 
scrutinized. Many Iterations may occur before the problem 
solver Is satisfied with the solution. When this state oc- 
curs, the solution Is prepared In final form and final test- 
ing Is per formed . 

If the original problem is so difficult that no trial 
solution comes Immediately to mind. It should be transformed 
Into a newproblem as follows; 
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a» Identify the problem as a particular case of a prob- 
lem whose solution Is known. 

b. Solve a specific Instance of the problem In the hope 
that It will lead to the general solution. 
Solution of this new problem proceeds as above. 

Computer programming Is a form of problem solving. Its 
solution paradigm Is called the development life cycle and 
is organized Into stages. There are many life cycle charts 
In the literature , [NBS76] [DOD77 ] but no one current chart 
Is appropriate for all views of the development process. The 
one given In Figure 1 is representative. During the re- 
quirements stage, the problem to be solved Is carefully de- 
fined. The design stage Is when general solutions are hy- 
pothesized and data and process structures are organized. 
During construction the program modules are coded and de- 
bugged . The modules are Integrated and the Inter fa ces de- 
bugged. Testing begins as the program Is exercised over the 
test data selected during this and earlier stages. The pro- 
gram is used and maintained during the final stages. 

Some significant differences are apparent between the 
problem solution paradigm described in the first paragraphs 
and the development life cycle described above. The life 
cycle activity is expressed as a straight line solution 
whereas the previous paradigm emphasized iteration toward a 
solution.. Anyone who hae ever programmed knows that itera- 
tion is essential. Iteration is the result of the verifica- 
tion process and error assessment. Unless the correct prob- 
lem is stated and the correct solution achieved at the first 
try, modification and iteration occur. 



Verification should accompany each stage of the 
d ^^vel o pment life cycle. If theverlficatlon process is iso- 
lated in a single stage then problem statement errors or 
design errors discovered at that stage may exact an exorbi- 
tant price. Not only must the original error be corrected 
but the structure built upon it must be changed also. 

Viewing each development stage as a sub-problem leads 
to a more productive paradigm for program development. The 
amend ed life c ycl e chart in Table 1 presents the verlfica- 
tion activities that accompany each development stage. 
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e 1. Life Cycle Verification Actlvltle 



! Tl'fo pTrr»l£» O *- ^ ^ ^ 

• Lt± JL e oyc 1 e b t„ag 6 


: Verification Activities 


: Requirements 


: Determine Correctness 

: Generate Functional Test Data 


: Design 


: Determine Correctness and Consistency \ 
Generate Structural and Functional : 
Test Data ! 


: Construction : 


i^etermine correctness and Consistency : 
Generate Structural and Functional : 
Test Data \ 
Apply Test Data \ 


: Operation : 
& Maintenance : 


Retest 



At each stage the verification activities should include: 

1. Determine the correctness and consistency of 
the structures produced at the stage, and 

2. Generate test data based upon structures in- 
troduced at that stage. 

For the design and construction stages it is also necessary 



3. Determine that the 
with those at the previous 

k . Refine and redefine 

Her. 



structures are consistent 
stage, and 

test sets generated ear- 
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Performing the above activities at each development 
stage should help to locate errois when they are Introduced 
and will also partition the test set construction. 

3. Testing 



Great strides have been made toward the development of 
formal verification techniques. These technl ques , based on 
ideas of formal semantics and proof techniques, while being 
promising research avenues are not easily applied without 
supporting tools (verifiers). Current 1 y automated verifiers 
are expensive, not widely available, and limited in applica- 
tion. For the single programmer, testing is the most easily 
applied verification technique. Testing is, as we will dis- 
cuss, limited in its ability to demonstrate correctness. 
Testing shows the presence of errors , and generally (ex- 
cluding exhaustive testing) cannot demonstrate the absence 
of errors. 

One view of a program is as a representation of a func- 
tion taking elements from one set (called the domain) and 
transforming them into elements of another set (called the 
range). The testing process is then used to ensure that the 
Implementation (the program) faithfully realizes the func- 
tion. Since programs are frequently given inputs that are 
not in the domain, they should also act reasonably on such 
elements. Thus a program which realizes the function 1/x 
should not fail cat as trophicall y when x=0 , but instead 
should generate an error message. We call elements within 
the functional domain val id inputs and those outside the 
domain inval id inputs. 

The goal of testing is to reveal errors not removed 
during debugging. The testing process consists of obtaining 
a valid value from within the functional domain or an In- 
valid value from outside the functional domain, determining 
the expected (correct) value, running the program on the 
given lvalue, observing the program*s behavior, and finally 
comparing that behavior with the expected (correct) 
behavior. If the comparison is successful , the result of 
the testing process has rev ea led no errors. If the compari- 
son is unsuccessful, then through the testing process the 
errors are revealed. 



The key words in the paragraph above are exyec t ed 
( correc t ) behavior , observati on , and comparison . An impor- 
tant phase of testing lies in planning how to apply test 
data, how to observe the results, and how to compare the 
results with desired behavior. Applying and observing tests 
are not always straightforward activities. Often extensive 
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analysis Is required to determine tests which adequately 
test design components, and often code must be Instrumented 
to provide o'b se rvat 1 on • Determining' the desired (correct) 
behavior for comparison with observed results Is very diffi- 
cult. To test a program, we must have an "oracle" to pro- 
vide the correct responses and, to represent the desired 
behavior. This Is a major role of a requirements specifica- 
tion. By providing a complete description of how the system 
Is to respond to its environment, a good requirements 
specification may form a basis for constructing such an ora- 
cle. In the future, executable requirements languages may 
provide this capability directly, but currently they are 
still basic research topics, consequently we must be content 
with more ad hoc techniques. Some typical ones include: 

1 . Int ui t i on . 

2. Hand calculation. 

3. Simulation, both manual and automated. 

4. An alternate solution to the same probletn. 

Although we have been discussing testing through exam- 
ples at the coding level, our Intent is to be general enough 
so that the discussion of the val Idat ion method s applies to 
any stage in the program's life cycle. Testing in its nar- 
rowest definition is performed during the construction 
stage, and then later during operar.lon and maintenance as 
revisions are made. Derivation of tesf data and test plan- 
ning, however, are activities which should cover the entire 
life cycle. 

If a broader meaning of testing is used, subsuming more 
of the verification process, then testing is an Important 
activity in each life cycle stage. Simplified walkthroughs, 
code reading, and most forms of requirements and design 
analysis can be thought of as testing procedures. Each of 
these wlir be discussed in later sections. 

The main problem with testing is that it reveals only 
the presence of errors. A complete validation of a program 
can be obtained only by testing for every element of the 
domain. Since this process is exhaustive, finding the pres- 
ence of no errors guarantees the validity of the program. 
This technique is the only dynamic analysis technique with 
this guarantee. Unfortunately, it is not practical. Fre- 
quently program domains are infinite or so large as to make 
the testing of each element of the domain Infeaslble. There 
is also the problem of deriving, for an exhaustive input 
set ; the expected (correct) responses. Such a task is at 
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least as difficult as ( and possibly equivalent to) writing 
the program Itself. The goal of a testing methodology Is to 
reduce the potentially Infinite exhaustive testing process 
to a finite testing process. This Is done by choosing 
representative elements to exercise features of the problem 
under solution or of the program written to solve the prob- 
lem. 

A subset of the domain used In a testing process Is 
called a test data set. Thus the crux of the testing problem 
Is finding an adequate test data set, one that covers the 
domain and yet Is small enough to use. This activity must 
be planned and carried out In each life cycle stage. Sample 
criteria for the selection of test data for test sets In-- 
elude : 

1. The test data should reflect special proper- 
ties of the domain such as extremal or ordering pro- 
perties or singularities. 

2. The test data should reflect special proper- 
ties of the function that the program Is supposed to 
Implement such as domain values leading to extremal 
function values. 

3. The test data should "exercise" tiie program 
In a specific manner, e.g., causing all branches to 
be executed or all statements to be executed. 

The properties that the test data sets are to reflect 
are classified according to whether they depend upon the 
program »s internal structure or the function the program is 
to perform. In the first two cases above, the test data re- 
flect functional properties and in the latter case structur- 
al properties. Structural testing helps to compensate for 
the Inability to do ^.xhaustive functional testing. 

While criteria for a test set to be adequate in a 
structural sense are often simple to state (such as branch 
coverage), the satisfaction of those criteria can usually 
only be determined by measurement. Due to the lack of 
analytical methods for deriving test data to satisfy struc- 
tural criteria, most strjiictural test sets are obtained using 
heuri s 1 1 c s . 

For Afunctional analysis techniques, the major cut'rent 
difficulties are in the specification of such vague terms as 
extremal or exceptional value. Further, for some functional 
analysis techniques, it may not be possible to obtain a 
functional description from a requirement or specification 
statement. Thus once again a substantial amount of effort in 
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test sec construction for functional considerations Is o? a 
heuristic nature. 

The general concepts of testing will be discussed below 
as they apply to each life cycle stage. Keep In mind 
throughout the discussions that planning Is the key to suc~ 
cessf ul testing . 

4, Requirements 



INVEST IN ANALYSIS AT THE 
BEGINNING OF' THE PROJECT 



Why bother with formal requirements for a small program 
written by a single programmer? The answer Is that an In- 
vestment in careful anal ys Is at the very beginning of the 
project can reap benefits even for the programmer working 
alone. Having a clear, concise statement of the problem to 
be solved will facilitate construction, communication, error 
analysis, and test* data generation. 

What should be included in the requirements or problem 
statement? Tha following list suggests the information that 
should be recorded and the decisions that should be made at 
this stage in the development. 

1. Functionally, what the program is to do. 

2. What the input will be like such as the 
• form , I'ormat , data types, and units for the input. 

3. The form, format, data types, and units for 
the out put . 

4. How exceptions, errors, and deviations are 
to be handl ed . 

5. For scientific com put at ions, the numerical 
method or at least the required accuracy of the 
sol ut 1 on . 

6. The computer environment required or as- 
sumed, e.g. the machine, operating system, and im- 
plementation language. 
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Making and recording decisions on the Issues listed 
above are only part of what should be done during the re- 
quirements stage of development. In addition, the' core of 
the test data set should be established and error analysis 
should be performed. 



The requirements state what tVie program Is to do. Data 
should be generated which will determine If the requirements 
have been met. To do this the Input domain should be partl- 
tloned Into classes of values that the program will treat In 
a similar fashion. For each class a representative element 
should be Included in the test data. Boundary values, ele- 
ments at the edge of the input class, frequently require 
special treatment by the program and are thus a likely 
source of error. Therefore, boundary values for each class 
should be Included in the test data set. In addition, any 
non-extremal input values that will require special handling 
by the program should be included In the test data set. Out- 
put classes should also be covered by input which causes 
output at each class boundary and within each class. In- 
valid input values require the same analysis provided for 
valid val ue s . 

For all elements in the test data set expected values 
should be determined. That is, how the program should treat 
each of the elements should be decided ana recorded. The 
test data set and accompanying expected values form the core 
of the test set that will be refined to use with the imple- 
mentation code. The test set generated during the require- 
ments stage will ad dress the functional aspects of the pro- 
gram. Data to test the structural aspects will be generated 
during the design and implementation stages. 

Generating test data at this stage also serves another 
useful purpose, that of insuring that the requirements are 
testable. Requirements for which it is impossible to define 
test data or to determine the expected value for the test 
data are ineffective requirements and should be reform ulat- 



START DEVELOPING THE TEST 
SET AT THE REQUIREMENTS STAGE 



ed. 
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ARE YOU SOLVING THE CORRECT 
PROBLEM? 



Generating the test set Is only part of the verifica- 
tion activity that should accompany the requirements stage. 
Analysis for the correctness, consistency , and completeness 
of the requirements should also be done. Whether the correct 
problem Is being solved should be considered. Perhaps a more 
general solution or a more specific solution would be more 
advantageous. The requirements should be carefully checked 
for conflicts and Inconsistencies. The possibility of miss- 
ing cases should be considered. Have functions been omitted 
that should have been Incl uded ? Too frequently the results 
of such analysis aren't obtained until the program Is exe- 
cuted with test data. At the Implementation stage the cost 
of requirements modification Is high and the rework Is 
painful. Therefore, error analysis early In the development 
cycle Is advocated. 

5. Design 



WRITE AN EXPLICIT DESIGN 
STATEMENT 



Even for small projects, the programmer shoul d spend 
the time and energy to produce an explicit statement of the 
design, for such a document will aid construction, communi- 
cation, error analysis, and test data generation. The re- 
quirements and design documents together should describe the 
problem to be solved and organization of the solution. They 
should conviey enough information so that the reader can 
determine what the program is to do and how it is to do it 
without having to resort to code reading. 

Several different design techniques and methodologies 
have received wide coverage recently [EDP79]. Whether you 
use one of those or your own technique, the following infor- 
mation should be included in the design. 

1. Principle data structures should be speci- 
fied since the way key data are organized frequently 
shape s the structure of the en tire program • 
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2. Functions, algorithms, heuristics, or spe- 
cial techniques used for processing should be 
recorded • 

3. The basic program organization should be 
stated. How the px^-^gram Is to be sub-dlvlded or 
modularized and the Internal aAd external Interfaces 
should be specified, 

4. Additional Information may be needed for 
particular projects. 

Verification activity Is very Important In the design 
stage. If faults are not found until after the program is 
:coded, fixes are often more costly and less satisfactory. 
VeTif ication at the design stage follows the. steps listed 
in section 2 • 

A. The design should be anal yzed to det erml ne 
If It Is complete and consistent. The Individual al- 
gorithms, heuristics, and special techniques should 
be checked to see if they really work for the given 
problevii and dat a , Thl s could Involve hand calcula- 
tl ons for re pre sent at Ive test cases. The total pro- 
cess should be analyzed to determine that no steps 
or special cases have been overlooked. The module 
Int erfaces should be examined to assure that calling 
routlnesprovldethe information and environment re- 
quired by the called routines and in the form, for- 
mat, and units assumed. The data structure should 
be examined for inconsistencies or awkwardness , 
Input/output handling should be carefully analyzed 
for it is a frequent source of error, 

B. The d e s 1 g n sh ould be analyzed to dete rm 1 ne 
If it satisfies the requir ement s . Determine if all 
constraints -jpecified by the requirements have been 
met. Determine If the design assumes the same form', 
format and units for input and out.put that are stat- 
ed in the requirements. Check that all- functions 
listed in the requirements have been included in the 
design. Selected test data generated during the re- 
quirements should be hand simulated to determine if 
the design would produce the expected values; 

C. Test data based upon the design should be 
generated .During the design stage data structures, 
functions, algorithms, heuristics, and the general 
program str ucture have all been established; data to 
test these structures and function/s should also be 
generated. Standard, extremal, and special values 
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for the data structures should be Included In the 
test data set. Boundary value analysis should be ap- 
plied to both the structure and the values of the 
data structures. For example, If array size can 
vary, then a single element array, a maximum size 
array, and special valued arrays should be candi- 
dates for the test data set. Test data for external- 
ly visible functions has been generated at the re- 
quirements stage. For the Internal functions Intro- 
duced during design, test data should also be gen- 
erated. The Input and output analysis suggested in 
Section 4 should be used to gesnerate the test data 
for these functions. If not already accomplished by 
the existing test data, new tests should be generat- 
ed to exercise the modular structure of the design. 
For all the data In the test data set, expected 
values should be calculated. The test data generat- 
ed during the design stage should test both the 
structure and the internal functions of the design. 

D. The test set generated during the require- 
ments stage should be re-examined In view of the 
design. Since additional decisions have been made 
during design, it Is possible that data and expected 
values generated during requirements can be refined 
and made more exact. 

^^^y P'^ogrammers and managers want to rush directly to 
the coding stage when they begin a new project. However, the 
time spent in careful analysis during the first two life cy- 
cle stages can Increase the quality of the total oroject. 
Following the above four steps will greatly facilitate pro- 
gram verification. Steps one and two should also be carried 
out by a colleague; specific techniques will be discussed in 
Section 6. It is so difficult to find your own errors that 
an Independent analysis is strongl y advocated . 

5. Construction 



Itie construction or coding stage is what most people 
think of When they think of programming. However, if the re- 
•quirsmsats -and design have been carefully done, the coding 
should be straightforward, almost mechanical . Since so much 
has been written about software engineering, structured pro- 
gramming, and various coding techniques, little will be said 
here except that we assume the programmer Is using good pro- 
gramming techniques [ZELK79]. 
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CHECK FOR CONSISTENCY 
WITH DESIGN 



The first step In verification during the construction 
stage Is to determine If the code Is consistent with the 
design. Both code and design should exhibit the same modular 
structure arid have the same module Interfaces. Both should 
utilize the same data structures and Impl ement the same 
functions using the same algorithms. Input / out put handling 
In the code should be consistent with the decisions made 
during the design stage. 



KEEP YOUR TESTS 



Testing should be performed In an organized and sys- 
tematic manner. Test runs should be dated, annotat ed , and 
saved. Not Infrequently testing Is slow and Ineffective be- 
cause It Is performed In a random manner. The programmer 
manufactures test data on the spot and runs tests as the 
spirit moves him using his memory as the only record of what 
has been prevlousl y test ed . Such random testing will produce 
random results. The systematic development of the test set 
through the re qui-rement s , design , and constr uct I on stages 
helps to produce an adequate test set; however, the actual 
execution of the program using the test data must also be 
performed in an orderly manner. A plan or schedule of which 
code pieces or modules are to be tested and in what order 
can be used as a check list to help the programmer organize 
his efforts. All test data and runs should be saved after 
being dated and suitably annotated. If errors are found and 
changes made to the program, retestlng must be performed. 
Not only must the given test be rerun, but also any tests 
previously run and passed that Involve the erroneous segment 
must be rerun since the program modifications may have in- 
validated the previous tests. 



ASK A COLLEAGUE 
FOR ASSISTANCE. 



During the construction stage, code exists and testing 
can begin. However, additional manual analysis of the code 
should precede the first test runs. Desk checking, in which 
a programmer sits at a desk reading his code, looking for 
errors, is not a particularly effective verification 
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technique. Usually programmers are poor at finding their own 
errors. Fundamental logic errors and missing cases elude 
the creator/programmer for If It seemed right when It was 
designed, It still seems right. As for syntactic errors, the 
programmer tends to see what was meant rather than what was 
cod ed • 



Inspection, an exercise In disciplined error hunting, 
and walk-through, a form of manual simulation, are formalized 
manual techniques based upon desk checking. In both tech- 
niques a team of programmers examine the code (or design or 
requirements) and look for errors In a very organized 
ma nne r • 

^"^^^l P'^°S"'"s. a technique partway between desk 
checking and Inspections or wal k-throughs seems appropriate. 
An Independent party, usually a colleague, should be asked 
to analyze the development product at each stage, the re- 
quirements, the design, and the code. The programmer should 
explain the product to his colleague, while the colleague 
plays devil s advocate, questioning the logic and searching 
for errors. A check list of likely errors should be used to 
guide the search. The second party aspect of this technique 
IS essential to its success for a new perspective is needed 
to locate the errors the programmer cannot perceive. 



USE AVAILABLE TOOLS 



u ,A l^^t the code is ready to be run. The programmer 
should be familiar with the various compilers and inter- 
preters available on his system for the language he is us- 
ing, if more than one exists, they undoubtedly differ in 
their error analysis and code generation capabilities. Some 
.processors do syntax checking only with no code generation: 
others perform extensive error checking such as array bound 
checking and provide aids such as cross reference tables. 
If debugging compilers exist on your system, it is very 
helpful to use them. 



Testing should exercise and stress the program struc- 
ture, the data structures, the Internal functions, and the 
externally visible functions or functionality. Both valid 
and Invalid data should be in the test set. The test set 
Which consists of test data and expected values generated 
during the requirements and design stages, forms the core of 
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the test set to be used during construction. Additional 
data may be required to test structural and functional de- 
tails visible only In the code. The test set generated dur- 
ing the first two life cycle stages may require refinement 
or redefinition in order to be applied to the code. The test 
data should be organized to conform with the general test 
organization to be used during construction. Intermediate 
values may be required in order to test Inco.i'^^ete pieces of 
code or individual modules. Testing requires as much 
creativity as does design. 



TEST ONE AT A TIME 



As with many tasks, divide and conquer is the rule for 
testing. Pieces of code, individual modules, and small col- 
lections of modules are exercised separately before they are 
integrated into the total program. Only after the pieces 
have been found error-free are they joined together, one at 
a time, and tested as a unit. It is easier to Isolate errors 
when the number of potential interactions is kept small. 

In order to test the smaller units of code, some addi- 
tional code may he required. If the testing is done bottom- 
up, then drivers will need to be written. For small progr^ims 
this may only entail producing code to call a procedure so 
the Interface and control transfer can be tested. If con- 
struction is proceeding top-down, then stubs will be needed. 
In this case incomplete or dummy called routines must be 
constructed so the flow of control and argument passage can 
be tested before the entire program has been constructed. 
Often there is a reluctance with small programs to want to 
generate any "unnecessary" code solely for testing. However, 
it is wise to organize the testing so that small pieces of 
code are exercised and deemed error-free before aggregating 
them into ever larger units. 

Instrumentation, the Insertion of code into the program 
solely to measure various program characteristics, can be 
useful for program verification. For medium and large scale 
projects tools are often acquired or developed which will do 
automatic Instrumentation . For small projects the programmer 
can do "a is own instrumentation. Array bound checks, checking 
of loop control variables, determining if key data values 
are within permissible ranges, tracing the execution, and 
counting the number of times a group of statements is exe- 
cuted are examples of the types of analysis that can be per- 
formed using instrumentation. 
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ARE YOU EVER THROUGH? 



How do you know when you have tested enough? That's a 
question that unfortunately haf no clllr\ut 



cute your program, then testrnglho^;; d\%:;r;u^::^L ^^^at^e^^ 
pfaJ narM^nr '° cluster, so those modules that Ip- 

tlnv vnn i """"^ ^^""l'^ receive special scru- 

you have run your complete test set against t h . 



answer Tf -n^^ ""fortunately has no clear cut 

answer. If you are^still finding errors everytlme 

. Ai 

2S 

spec 

progra. and have found ' no-mo";;'';;;o%s d^e^^ noi'mea'n tJat 

your program Is error free. Perhaps your test s^t 1^' ?n om^ 
rl^lV Retries that tend to be used to measure testing 

thoroughness Include: statement testing, branch testing and 

?n ^h 'p^ojfam ^L'^r"' testing determines If each stafem^n 
Jna ^ program has been executed at least once. Branch test- 
ing determines If each exit from each branch In the progrL 
has been executed at least once. Path testing determines i? 
all logical paths through the program, which may Involve r^- 
Ha t TnlT'r segments, have been'exe^^ L "t 

^^ u^""^ Succeeding metric subsumes the others, 
numhp/ ."""^ °^ P^'^' S"^^ exponentially with the 

costly if tlm: %'°" P^^' '^^''^^ '^-'^ be too 

costly In time and money to utilize and often Impossible to 

insufflcle^r^l's'tn''"' ' ^° theoretically 

insutticlent , Is the coverage metric most frequently used 
because It Is re 1 at i v el y s Impl e to Impl ement . ''"'^"''^ ""^'^ 

If dynamic analysis tool that measures test coverae-e 
c se'to :s^ °" ^ ^'--^^^^ forward exer- 

coverlge IdenM?v '° calculate statement 

coverage. Identify each program point into which control can 
be transferred and establish an array with as many slo^s as 
Irtr points. At each transfer point nsert a 

arrar'^hf will increment the approprlat e'slot ?n the 
array. The final values in the array can be printed at pro- 
t": '-tesJ' technique will not only provide af l^d^c^"- 
tion of test coverage but will also show the portions of 
your program with the heaviest usage. Statement testing will 
not guarantee your program is correct; however it is a 
testing minimum that .s frequently used fo^ if lole'dl nev- 
free exercised. it is difficult to know if it is error 

^l.n.nT ^b°"lf be emphasized that the amount of testing will 
s^gmentsT^l r' °' .^"^ * ^-^^^-1 Programs or lode 

nlir^^t VuilllTs'llr^'^ testing^ than more Inslg- 
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7 • Oper a tl on 



Why talk about testing during operation? For many pro- 
duction programs 50 to 85% of the total life cycle costs are 
accrued during the maintenance stage that Is coincident with 
operation. These costs do not imply error studded programs 
that take forever to fix but evolving programs that are con- 
stantly undergoing modification. Even for small programs 
corrections, modifications, and extensions are bound to oc- 
cur. Any time there Is modification or change, testing Is 
required. Testing during maintenance Is termed regression 
testing. The systematic approach recommended In the rest of 
the report facilitates regression testing since much of the 
work should already have been done. The test set, test plan, 
and test results for the original program should exist. 
Modification to accommodate the program changes must be done 
and then all portions of the program affected by the modifi- 
cations must be retested. After regression testing Is com- 
plete, the program and test documentation must be updated to 
reflect the changes. To Ignore the need to maintain con- 
sistent and current documentation on both programs and com- 
pleted tests Is to Insure Increasing Instability of the pro- 
grams at each successive modification. 

8. Summary and General Rules 

We have proposed a general approach to developing 
software In an environment with limited resources. This ap- 
proach stresses that verification must take place throughout 
the development process. Even for relatively small., non- 
critical projects lifecycle planning for software quality is 
very important. A programmer's task of demonstrating that he 
has produced reliable and consistent code can be much sim- 
plified by beginning early to develop and plan verification 
activities. 

The approach we recommend relies heavily on testing as 
the main verification technique with code reading and in- 
spection techniques employed as supporting activities. Test- 
ing cannot be performed as an after-the-fact activity. Test 
data sets are derived throughout the development process be- 
ginning at the requirements stage. It is at the first stage 
that test data can be derived from functional analysis of 
the system requirements. Inconsistency in the requirements 
is often detected during this analysis. During the design 
stage, test data which stress the prog ram control and data 
structures are generated. When the final code generation 
and construction stage begins, the test data set should be^ 
nearly complete. The remaining test data should be chosen to 
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exercise those "special functions" used by the programmer In 
the actual Implementation. Care should be taken to Insure 
that the data and control structures represented by the fi- 
nal programs are covered as extensively as possible hy the 
t est dat a . 



The actual testing should proceed In an organized 
manner with small pieces being tested Individually, then ag- 
gregated and retested. Test coverage metrics are very useful 
for determining a "confidence level" for the test data. Such 
metrics can be Implemented by Instrumenting your own code. 
Since testing Is a process aimed at discovering errors, It 
must be noted, that for each design or code modification 
resulting from such a discovery, retestlng Is necessary/. The 
process becomes an iterative one towards a goal of achieving 
a high level of confidence In the finished product through 
intelligent testing. ^ 

Inspection, close reading and analysis of the code, and 
manual simulation are important verification techniques 
which should also be used throughout development. Thorough 
analysis and hand simulation are employed during both re- 
quirements and design in order to discover errors early in 
the development. Each stage must be analyzed for consisten- 
cy with prior stages and for internal consistency. You 
should seek Independent review of your work by having a col- 
league Inspect your design and code for errors. We believe 
that quality can be improved through planning, a staged 
development, and the use of sound verification techniques at 
each stage. 

The following list summarizes the approach presented in 
this report. 

1. Generate test data at all stages and in par- 
ticular the early stages. 

2. Develop a means for calculating the expected 
values for test data to compare with test results. 

3. Inspect requirements, design, and code for 
errors and for consistency. 

4. Be systematic in your approach to testing. 

5. Test pieces and then aggregate. 



6. Save, organize, and annotate test runs. 



7. Concentrate testing on modules that exhibit 
the most errors and on their Interfaces. 

8. Re test when modifications are made. 

9. Discover and use available tools on your 
system. 
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